A way and apparatus that may be employed to efficiently cool data centers and electronics housed in enclosures that employs Loop Heat Pipes and other passive technologies to cool the primary heat loads in such systems and which employ a combination of other methods to cool the secondary heat loads.
The cooling of electronics housed in enclosures has for many years been dominated by methods that were more concerned about getting the job done than the energy it took. Methods for improving the efficiency of electronic cooling using passive heat transfer such as heat pipes have been available since the Manhattan project, yet have only become inexpensive with the advent of CPUs that rejected 40 or more Watts, and needed to extend the operating capabilities of finned heat sinks. This disclosure employs passive closed loop heat transfer devices that can dramatically improve not only the energy efficiency of electronic cooling but also makes it possible to cool devices that reject 500 or more Watts mounted on densely packed printed circuit boards and in the case of a data centers, reduce the energy required to cool it by 80% or more!
The devices at the heart of this disclosure are Loop Heat Pipes, Capillary Pumped Loops and derivatives of Loop Heat Pipes that include devices like pumps in the condenser lines. We lump all these devices together into the category Loop Heat Pipe Like (LHPL).
In general the LHPLs employed provide the best energy efficiency of any electronic cooling device ever invented. Not only are they passive, but their ability to reject heat to new locations is often measured in meters employing very small condenser pipes (often less than 3 mm) that make it possible for them to move heat out of tight spaces to condensers that can very efficiently reject heat employing heat transfer devices that have large contact areas to secondary coolants such as air and water. LHPLs minimize the energy required to reject energy, but they also maximize the quality of the heat being passed to their secondary coolant streams. It is this second attribute that often just as important as the first, when it comes to producing systems whose overall energy efficiency has been maximized. It is their ability to transport heat to new locations in a chassis or enclosures and then pass it through a reasonably long small pipe to a condenser that distributes it over a reasonably large area, that makes it possible to efficiently transfer the heat being rejected directly to a secondary coolant such as chilled water that results in the high quality heat that make it possible to transport the heat leaving rack mount servers directly to devices cooling towers that make these devices so valuable in data center cooling. The secondary coolants that receive this heat end up with the highest delta T's of any primary heat load rejection technology that we know of.
It is the low total thermal resistances of 0.15° C./Watt and heat transfer coefficient of 0.15° C./(Wcm2) that make this outstanding performance possible. The state of the art Loop Heat Pipes employed in this disclosure employed devices disclosed in the past that people familiar with the art know how to make and were made of Nickel with stainless steel condenser pipes or copper and employed Ammonia and Water working fluids. The evaporators of these devices were only 0.3 inches tall and mounted on the heat spreaders of CPUs were small enough to easily fit on PCBs set on 0.8 inch centers. In the case of a 100 Watt CPU whose LHP condenser was cooled by 30° C. water and whose condenser produced 47° C. water, the CPU being cooled reported a die temperature of just 62° C. Translated into practical terms, employed in the 1 U rack mount chassis that dominate modern data centers, this device makes it possible to remove the heat directly from a server to the data center's cooling tower or other heat sink. In the process, the noisy fans in the racks that add numerous points of failure and consume as much as 30% of the server's power along with the CRAC unit's air blowers and water chillers (that together consume 35% of the data center's total power) can be eliminated! The 30° C. temperature chosen came from the ASHRAE “0.4%” tables for Atlanta Ga. —this was the worst case cooling effluent that could be expected from a commercially available evaporative cooling tower at that location. A quick comparison of the power consumption at institutions like Lawrence Livermore National Labs suggests that:
Electronics50%Water Chiller25%Air Blower10%1U fans9%UPS5%Lighting1%changes to this:
Electronics 83%1U fans1.6%Cooling TowerPump and fan  5%UPS8.3%Lighting1.6%Which is to say the total power consumed by the data center goes down by 40%!
The same energy benefits that accrue to data center cooling also accrue to the general cooling of all electronic enclosures that are air cooled, but to a lesser extent, for the simple reason that air is a much poorer heat transport medium than the chilled water. At the head of the list of benefits in addition to reduced energy costs are huge reductions in noise, the elimination of heat arriving at the walls of enclosures that can be so hot that it is almost possible to get burned touching them, the frequent failure of rotating cooling components including fans and pumps (in the case of pumped liquid cooling) which now occur so often that the systems that employ them have to mount them so that they can be easily swapped out without turning the machine off along with the ability to reject heat loads from devices that produced 500 or more Watts and to cool efficient devices such as CPUs and GPUs mounted in laptops where improved energy efficiency can improve battery life.
To appreciate the benefits of employing LHPLs to cool electronic enclosures, including air and water cooled electronic devices used to do everything from space vehicles to the rejection of heat to the cooling towers of data centers, it is first necessary to consider the goals of this disclosure. These are to efficiently cool electronic enclosures that contain semiconductor devices that may reject 200 or more Watts in the future and often end up being mounted on densely packed PCBs along with the devices that support them and reject much less heat. And, to carry out this task while maximizing the quality of the heat being rejected to the cooling systems providing the coolants being employed, as much as possible. In the case of air cooled enclosures housed in rack mounted chassis, we wanted to make both efficiency and maximization of heat quality achievable at the same time which can dramatically reduce the cost of cooling serves that continue to be housed in air cooled data centers. In cases where liquid cooling including chilled water is available, our goal was to reject high quality heat all the way back to the data center cooling tower, on a year round basis in most localities in the world. To achieve these goals, we employed LHPLs, some of whose other outstanding properties ignoring the fact they consume no energy turns out to be that they eliminate most of the electric motors, fans, blowers, compressors and other rotating devices found in servers and throughout the data center that end up making noise, contribute to frequent server failures and cost money to maintain and operate.
These goals don't get met without skepticism from practitioners of the art that employ alternative and competitive technologies, so we will now address the advantages of our approach in detail, while at the same time laying out the critical items that need to be overcome to reach our goals.
There are a large number of inventions that have been recently been investigated whose main purpose has been to improve the heat transfer capabilities of devices that can be used to cool semiconductor devices that reject large quantities of heat. LHPLs continue to provide as good or better performance as any of these other devices while at the same time reducing total energy expenditures. The highest performance competitors that we are aware of that actually work and compete with passive two phase solutions typically employ sensible cooling, including microchannels and jet impingement both of which employ liquids pumped under pressure. Microchannels drive the liquid across a heat spreader whose contact area with the heat source being rejected is increased using serpentine shaped channels. The porous metallic wicks employed by LHPLs can provide total contact areas between the working fluid and the heat coming into the wick that are thousands of times greater than the cross sectional areas of the dies being cooled. It is difficult to achieve the same surface area ratios using microchannels and at the same time provide uniform cooling across the die. In addition, it is difficult in general for sensibly cooled devices to make up for the 100 to 1 increase in cooling performance per gram of working fluid that phase change technologies provide. The other problem with either approach is that It does not provide the counter-flow heat transfer principles that LHPL wicks make possible (i.e. in microchannels the flow is across the die and dies tend to be hot everywhere, whereas the flow in a wick starts at the cold end away from the die and progresses towards it hot end getting hotter as it gets closer to the heat source (uniformly across the die) until the working fluid finally reaches the temperature where it boils, at which point it exits stage right through a channel designed to carry it off leaving the liquid that has not reached boiling behind and not mixing with it: what leaves for the condenser is hot gas and nothing but hot gas.
Jet impingement technology on the other hand ought to provide uniform cooling but suffers from another problem, which is mixing of hot and cold liquid in the cavity where the heat exchange between the die (or its heat spreader) go on in each of many such cavities. This mixing is an invariable result of the fact that to get good transfer between the surface being cooled and the working fluid turbulent flows need to be employed. The heat transfer coefficients measured using this technique like the former can be very high, simply because they do not take into account the quantity of coolant that needs to be pumped across the processor. In our case, the water flow rate required to transfer 100 Watts turns out to be just 1.2 ml/sec, which is one of the reasons that we frequently measured delta T's as high as 17° C. when cooling with our ideal secondary coolant, water. In either event the two most crucial attributes that get solved by LHPLs when it comes to problem of things like global warming are the efficient removal of energy from the CPUs and the retransmission of this energy to a secondary coolant without making a major reduction of the quality of the heat being rejected by the CPU.
One of the big problems in energy conservation is not understanding the important role that the quality of the heat being rejected to the final cooling device in major thermodynamic systems plays in the overall cost of buying and operating such systems. In the case of studies done for DOE on employing an Ammonia Rankine Bottom Cycle to increase the energy efficiency of a 12 megawatt fuel cell power plant it was discovered that extracting too much energy from the exhaust flow of the fuel cell ended up driving up the cost of the energy required to run the cooling fan in the air cooling tower, limiting the account of energy that could be saved to around 8% but adding a five year payback on the device that improved plants efficiency. The implication for data center cooling is, keep the quality of the heat up, unless you want to spend a lot of money to reject it at the cooling tower. In existing data centers the buck gets passed to the HVAC guys whose job it is to sell uses of large enough water chiller to raise the temperature of the CRAC unit's output so it can drive a cooling tower. This consumes 25% of the total energy budget of the data center.
The naïve approach to the use of passive heat devices suggests that like the extra cooling loops that currently consume close to 35% of the energy required to run a data center, not understanding how to apply them, can result in similar losses. It is quite easy to show, for example, that a sequential series of passive heat transfer devices passes less energy along than a single efficient passive device for the simple reason that energy losses in passive devices are dominated by the thermal resistance at the end points and any sequence of such devices has more such end points. It also turns out that in the case of optimizing the energy consumption of a data center cooling system, the crucial role played by passive devices turns out to be moving the energy as quickly (i.e. using the shortest length of condenser tubing) as possible to the best possible device that can move it the long distance it needs to travel to get to the cooling tower suffering the smallest reductions in heat quality. And the device that does the best job at this last task is called an insulated pipe and what runs down that pipe most often turns out to be water. It turns out moving megawatts of energy from the data center floor to the cooling tower simply is not practical for the same reason that moving the hot air in a data center room back to the cooling tower is not. So by definition one of our ultimate tasks is now well defined, create the LHPLs required to remove the primary heat loads using the most efficient condensers and evaporator to CPU interfaces that can be made, and in the case of a secondary coolant that happens to be chilled water, interface them to this supply using the shortest possible condenser lines so that the temperature of the water being produced is as high as possible. And the secondary task turns out to be, transfer the remainder of the energy in the system to chilled water as well, employing the smallest amount of power.
This last paragraph in a nutshell contains our marching orders. And, the two crucial things that we need to do to carry out this set of commands when it comes to the use of LHPLs, is remove the heat from the die being cooled wasting the smallest amount of heat. And, when the vapor the LHPL has created ends up at a condenser, make sure that the condenser gets its job done wasting the smallest amount of heat while at the same time maximizing the quality of the heat being transferred. The first chore is done using heat spreaders that mate to the CPUs we need to cool and don't waste any heat depositing it in the LHPL's evaporator wick. The second chore is designing condensers that waste as little energy as possible and when it comes time to cool with liquids, employ efficient counter-flow techniques to the problem that maximize the quality of the heat being rejected. While it sounds simple, the majority of the prior art that we have visited ignores the problem of maximizing the quality of the heat, which is not all that surprising, we did as well at the start of our research until we remembered the lessons learned designing fuel cell power plants and figured out how they applied to efficiently rejecting heat in electronics housed in buildings and data centers.
Similar naïve concepts about the best way to employ Loop Heat Pipes and their derivatives to employing air to cool electronic enclosures that contain components rejecting large quantities of heat have also been around for quite some time. The first set of claims in this disclosure apply to air cooled enclosures only and use the concept that the LHPL condensers save the largest amount of energy in an air cooled enclosure when they are placed at points in the enclosure where the cooling air leaves the enclosure. In many instances this turns out to be the spot where the chassis's exhaust cooling fans are located. Placing LHPL condensers at these locations results in a reduction of the ambient temperature of the air in the chassis which in turn reduces the air flow rates needed to cool components cooled by circulating ambient air as well as the elimination of fans on the CPUs and other hot components cooled by the LHPL evaporators. In situations where there are many rows of hot devices the elimination of non-uniform cooling and the attendant reduction in flow speeds is great enough to make it possible for the fans that perform the chassis exhaust cooling to simultaneously cool the LHPL condensers, while at the same time pulling much less total air through the chassis which results in a large reduction in the energy needed to employ air cooling to cool electronics and dramatic reductions in side effects like noise.
While Loop Heat Pipes have been around for many years, the assumption of the people with prior art is simply that just employing passive heat transfer devices must end up reducing the cost of rejecting energy. While this is true, because you are eliminating fans, what they are ignoring is the fact that proper condenser design and placement can often double the energy savings.
It turns out that 30 to 40% of the energy rejected by servers comes from ancillary components, so we will also have to worry about minimizing these energy costs as well. In data centers in which the average rack cabinet only consumed 5 KW, the fans on the rear of rack cabinets were a convenient way to help cool the contents. However, their main function at today's power levels of 20+KW, is mostly to hide the unsightly cables that connect them. A significant portion of the air being drawn through a typical rack cabinet ends up being drawn around the stack of server chassis within it and often the asymmetric flows within the cabinet can result in eddies that circle back to the front of the cabinet near the top, heat up the top servers by as much as 15 degrees F. To get around that problem fans can be added to the top of the rack cabinet and baffles inserted between the servers and the side panels. A better way to employ such fans, is simply to insert a duct in the cabinet that can be used to gather up all the air from the rack mounted chassis and exhaust it out the rear of the cabinet by connecting it the fans on the rear door or out the top using fans mounted on the top panel or possibly to the CRAC units return air flow ducting. To make sure that this duct does what it is intended to do, a mechanism has been provided in the disclosure to seal the chassis to the duct and at the same time make sure that in the event that a chassis is not installed the duct does not leak. Furthermore, to help solve the problem of potential leaks in situations where direct chilled water is being employed within the rack cabinets, the duct can be used to contain the chilled water manifolds that serve the rack mount chassis. Finally, to make it possible for the air being removed from the rack to be reused without having to make the long trip back to the CRAC units blower, simply inserting chilled water air heat exchangers in the exit path from the rack mount chassis to the duct, makes it possible to eject the air from the cabinet at the ambient air temperature of the room. This strategy has a number of benefits that other approaches to the cooling of high power rack mount chassis that employ water cooled air heat exchangers within the rack cabinet do not. Besides taking up much less space in the rack cabinet, and making it possible to employ distributed heat exchangers whose total area is much larger than the ones employed by other solutions, it also reduces the total high speed fetch that the air has to make. And in the process, the amount of energy that gets injected into air flows ends up being minimized.
Reducing the energy employed moving air is one of our overall goals. When we have to do it, our goal is to move the air the smallest distance at the smallest possible speed that gets the cooling job done. The reason for this is quite simple, energy losses due to drag do not scale linearly with velocity, but at a faster rate. Keeping the velocity and distance down, makes an enormous difference in the energy consumed by the fans driving the serves, rack cabinets and the data center itself. This can be simply verified by examining the power required to operate 1 U fans. The earlier devices that turned at 10,500 RPM consumed one to two Watts and provided 12 CFM. The latest fans provide twice the cooling flow rate but consume three to four times as much energy, with the latest motor devices consuming as much as 8 times the power. The technology we employ cuts down on these losses three different ways. First, when exchanging energy between air and either the primary coolant being chilled in a condenser, or a chilled liquid that is cooling it, our methods employ finned condensers that have large areas, which the small size of LHPL evaporators enables by doing things like moving the heat being rejected out of the tight spaces between PCBs where there is no room to place large heat exchangers which in turn ends up reducing the velocity of the air required. Next, by moving air the smallest distances possible. Often this means going to rack mount chassis like 1 U chassis in which the tops and bottoms end up behaving like airflow baffles. While not necessary for our technology to function, the choice of the right chassis architecture can impact the overall efficiency of the cooling being provided. The other technique we employ is to cut down on the distance that air needs to flow at high velocities when it is being employed by enclosures housed in rack cabinets. One of our embodiments employs a negative pressure duct that is sealed to the chassis it is evacuating air out of and which can completely eliminating the need for the air to travel back to the data center's air heat exchanger making large reductions in the amount of energy that needs to employed by air fans and blowers. When compared with other techniques for cooling rack mount chassis housed in rack cabinets this embodiment ends up reducing energy by reducing the distance that the air has to move at high velocities while at the same time eliminating the need in many instances for the use of hot and cold isles. When this embodiment is coupled with the use of liquid cooled air heat exchangers mounted at the exit point of rack mount chassis, it makes it possible for air to simply leave the rack quickly, and then proceed across the isle to the next row of rack cabinet inputs, minimizing the total distance the air needs to be routed through high speed ducts and also reducing things like losses in ducts due to other problems like poor sealing. This strategy plays an important role in our energy conservation effort, and is embodied in both our sealed chassis and sealed duct designs.
The final energy reduction principle that needs to be taken into account that out embodiment improves is water condensation. In some data centers, as much as 40% of the energy being employed by water chillers gets used to remove (by condensation) water vapor from the cooling flow which then, apparently needs to get added back into the flow to keep IT people wandering through the data center happy. It turns out that there no longer is and ESD requirement on the minimum air content of the air being employed in data centers, which basically means that keeping the relative humidity below the point where condensation occurs in the equipment can now be achieved by simply making sure the dew point of the air in the systems being cooled is less than the temperature of the liquid coolant being employed to cool systems, saving roughly 10% of the energy employed to cool some data centers, especially those in humid localities.
Our sealed chassis embodiments make this possible by keeping the dew point of the air inside of the rack mount chassis below the temperature of the coldest liquid coolant employed. This is simply accomplished in an embodiment in which we pass slightly pressurized air through a cold trap that removes excess water from it before slowly bleeding it into the “sealed” chassis, that are allowed to slowly leak air back to the ambient, at a rate that guarantees that the average air content of the chassis remains dry enough to avoid condensing if and when it comes into contact with chilled surfaces.
When it comes to cooling air cooled enclosures in general, LHPLs make it possible to make great strides in efficient uniformly distributed air cooling, by the simple act of placing the LHPL condenser at the point in the chassis where the air flow is normally exhausted out of the chassis. In the two enclosures we have studied, 1 U rack mount chassis and desktop cooled chassis, the fans that are employed on the exterior surfaces of these chassis have provided high enough flow rates to in the case of a 1 U chassis only require a single blower (already employed to pull air out of the chassis) to cool a pair of 120 Watt processors (it normally takes four to eight 1 U fans to accomplish the same task) and a single 120 mm fan running at just 1800 RPM to cool a 500 Watt CPU sitting in either the PCIe bus of the system. In all of the chassis we have examined, including the 4 U chassis employed to cool four to eight Opteron multi-core processors, the existing fans on the rear wall of the chassis that we have examined have more than enough cooling fans to make it possible to cool all of the processors, without the need for CPU fans. Which is to say, all of the chassis tested, when their CPUs were cooled using LHPLs, could get by without the need for CPU cooling fans. Not only that, the CPUs that were being cooling in situations like the 4 P/8 P chassis, normally require very high air flow rates even with cooling fans that fit into 2 U tall spaces simply because the CPUs in the front row end up heating the air used to cool the rear row of processors. This problem goes away with LHPLs, making it possible to actually reduce the air flow rates on the rear wall while at the same time eliminating the four to eight fans typically used to cool processors. And, while we can't claim that air cooling does as good a job as water cooling, we have gone about as far as you can go with air cooling to maintain the quality of the heat being rejected. In addition to providing sealed ducts, more uniform distribution of cooling air across the chassis and the reduction of the ambient temperature within the chassis, we have also introduced LHPL condenser designs which employ counter-flow cooling, which results in increased exit flow air temperatures which in turn end up improving the efficiency of an air cooled data center's water chiller.
As a general principle then, one of our goals is to wherever possible, remove the heat being rejected by the hottest components in an enclosure and then deposit it someplace where it can be ejected to the next cooling loop, which keeps it from circling around and haunting us. The duct above helps make this possible by gathering it up and shipping it off to the next cooling loop. When installed in poorly designed rack cabinets or data center rooms, up to 20% of the heat that is being rejected will often end up congregating in high spots, and then come back to haunt the top most chassis in a rack mount enclosure. The other major problem we face is the secondary components that we have created embodiments to handle, which gather up their heat using the smallest amount of energy.
One major gray area here is DIMM modules which can often reject a surprising amount of heat, that needs to be rejected. Fully buffered DIMM modules can now reject up to 15 Watts each, which means an array of 16 such modules can reject 240 Watts, classifying this group of devices as a hot electronic device. While the industry is getting away from this particular type of module, there is no guarantee that in the future such devices will not reappear. One solution if there are not too many, is to simply cut a Nomex (a thin card board like material that is fire proof) baffle in the form of a channel, that fits over them, attach to the motherboard with adhesive to form a duct and then pull air vigorously through the channel. This works better than so called DIMM module coolers. We have included a FIG. 35) which demonstrates how to make a metal channel out of aluminum or copper than can fit over DIMM modules that have been equipped with heat spreaders, that removes the heat from the modules making it possible to distribute that heat into the motherboard they are mounted on. This device also will work well with blade solutions (see FIG. 21) where we suggest using this device and adding a heat pipe to it that then interfaces the cold spreader at the edge of the card using a simple device such as a stand off and screw that brings the heat pipe's condenser end into thermal contact with the card's cold spreader. In extreme situations where there are more than four DIMM modules being cooled the heat pipe could actually be replaced with a miniature LHP similar to the ones employed to cool CPUs.
When it comes to liquid cooling, the embodiments provided make it possible to employ LHPL cooling with condensers that are either directly or indirectly cooled with chilled liquids including water, safely. A new method for interfacing all closed loop passive heat transfer devices to chilled liquids has been introduced which employs a cold plate along with what we call a cold spreader (that is thermally attached to the LHPL working fluid's condenser lines) that comes into contact with the cold plate when a rack mount chassis gets installed inside of a rack cabinet. This interface, while not quite as efficient as the directly cooled interface we are about to describe, in certain situations, like blade and COTS Single Board Computer (SBC) situations, makes it possible to cool these devices as well, without using the quick disconnects that direct chilled liquids require. To improve the quality of the heat being rejected by these split condensers, a counter-flow version is also embodied and examples are provided of how to employ the cold plates that are a component of a split condenser to also cool air that is either circulating within a sealed chassis or being passed through a chassis that is being evacuated either by internal fans or a negative pressure air duct.
The majority of the world's data centers today employ air cooling and water chillers. Last year, when asked if they needed water on the data center floor, a majority of data center users said no. This year, the tide is changing and a majority now say they need it, especially in places like Europe where there is no excess capacity available on the grid to power data centers. As data centers shift to water on the floor cooling, they will be able to take advantage of the most efficient physical methods which this disclosure employs. These methods employ LHPLs that are directly cooled with chilled liquids whose components are housed in a sealed enclosure in which the remainder of the components within the enclosure are being cooled by either liquid cooled cold plates or by a combination of them and air that is circulating about a chassis or by a chassis that has access to chilled liquid cooled air heat exchanger—any combination of the three work equally well although cooling things like power supplies which have concentrated loads is always a benefit. In situations where there is a large content of cold plates the area within the chassis that can be classified as cold is huge making it possible for slow moving fans to actually out perform liquid chilled air heat exchangers. The precise combination is clearly a function of the components housed in any particular enclosure that is being cooled.
One of the most crucial aspects of any system whose goal it is to produce secondary coolants whose temperature is hot enough to make it all the way back to the cooling tower of a data center is the design employed in the LHPL chilled water condenser. The condenser design that we created that did the best job of producing high temperature effluent employed counter-flow heat exchange and used a chilled water jacket that was made of a material that does not readily conduct heat in addition to employing a helical wire that was thermally attached to the serpentine shaped condenser pipe, forcing the liquid to take a longer path and simultaneously increasing turbulent flow within the channel that improved heat transport between the liquid and the jacket that contained.
The final claim in the disclosure is for a data center cooled with the afore mentioned devices in which the servers in the data center room is directly attached to the cooling tower, eliminating the need for air ducting, special insulation in the walls of the data center (to keep humid air out), the need for an air blower and finally the water chiller employed by the air blower, in localities in the United States, when on the hottest most humid days of the year, an evaporative cooling tower will return water to the data center room that is at least 30 C, which is to say for most locations as hot and humid as locations like Atlanta Ga., 365 days of the year.